Monday, December 31, 2018

Implementing A Very Simple Weather Service - Part 1

I've got this travel hack in mind that depends on a really simple weather API. Specifically, I need a web service that will take in a date, forecast attribute (say, high temperature for the day) and a location and return back the single value that was asked for. From what I can tell, such a simple weather API doesn't exist. But it also shouldn't be hard to build one.

My plan is simple: use an existing weather API (like DarkSky's) and write a wrapper around it that makes it behave like the simple service I have in mind.

The first step in implementing my super simple weather service has nothing to do with the weather; it has to do with the location aspect of the code. Specifically, I need to turn human friendly addresses into a latitude, longitude pairs. This is known as Geocoding and Google Offers an API to do just this.

Here's a snippet of the code I've written that does geocoding:

function geocode($location, $options = []) {
  $info = curl_get('https://maps.googleapis.com/maps/api/geocode/json', [
    'address' => $location,
    'key'     => GEOCODING_API_KEY,
  ], ['ttl' => 60 * 60 * 25 * 365 * 10]);
  
  if(g($options, 'debug')) {
    var_dump($location);
    var_dump($info);
  }
  if(is_array($info)) {
    return g($info, ['results', 0, 'geometry', 'location']);
  } else {
    return false;
  }
}

And here's the curl_get function that powers the above code:

function curl_get($url, $params, $options = []) {
  $ttl = g($options, 'ttl', 0);
  $cache_file = CURL_CACHE_DIR . "/" . md5($url . serialize($params)) . '.json';
  if(file_exists($cache_file) && ((time() - filemtime($cache_file)) < $ttl)) {
    $body = file_get_contents($cache_file);
    return json_decode($body, true);
  }
  
  $fd = fopen($cache_file, "w");
  $ch = curl_init($url . ($params ? ('?' . http_build_query($params, false, '&')) : ''));
  curl_setopt($ch, CURLOPT_TIMEOUT, g($options, 'timeout', 4));
  curl_setopt($ch, CURLOPT_FILE, $fd);
  curl_exec($ch);
  $info = curl_getinfo($ch);
  if($info['http_code'] == 200) {
    $body = file_get_contents($cache_file);
    return json_decode($body, true);
  } else {
    unlink($cache_file);
    return $ch;
  }
}

There's not much to it, but I did implement support for caching. The idea is that once I've geocoded a location I shouldn't need to geocode that location ever again. Note that I setup curl_get's caching to support a time-to-live (TTL) parameter. This should be useful for when I re-purpose curl_get to implement the call to the underlying weather API. For these calls, I expect I'll set the TTL to say 6 hours, to allow the system to get refreshed forecast values yet still make aggressive use of caching.

You can view the source code for this project here. If you're inclined, you can mess with an instance of the code here, though there really isn't much to see yet.

Stay tuned, as I've got plans for finishing up this API soon.

Thursday, December 27, 2018

Lessons from a Fiery YouTube Comment | Thoughts On Using Emergency Gear

I found this Review / walk-through of the Hidden Woodsman Signal Panel to be informative. At the very least I appreciated that someone took the time to get out and use this piece of gear, rather than do a table-top review. After watching the video I scrolled down to read the comments and was taken aback by the first one:

There's NOTHING "common sense" about this video. An emergency SIGNAL panel should never be pulled out or used for ANYTHING other than to SIGNAL someone in the event of an emergency. Using it for anything else subjects it to dirt, grime, sunlight and all the other crap that will fade it's color and brightness ...thus inhabiting its effectiveness when you NEED to SIGNAL someone in the event of an emergency. There's about six of you idiots on YouTube who think it's cute to use your SIGNAL panel for everything other than what's its supposed to be used for. While you're trying to show off on here ALL YOU'RE DOING is putting out reckless ideas that ultimately could hamper someone's ability to get rescued in an emergency. Leave survival tips to the experts. We're all better off if you just stay home and eat your granola bars on the couch while watching Rambo. SERIOUSLY.

Ouch.

As I thought about this comment I found myself getting more and more fired up. Sure, part of what bothered me was the heavy handed way this individual delivered his (or her?) criticism. But more than that, I think his critique is as far off as he thinks the original video is.

I suppose it's technically possible that you'll find yourself in an emergency where you pull out your signal panel, only to find it worn and faded to the point where rescuers can't see it. But I think a more likely scenario is that you pull the panel from your pack only to find that it doesn't function as you imagined. Huh, you may think: I thought it was bigger, sturdier, came with guy-line tie-outs, was reflective, or any number of other attributes you imagined to be so but aren't. And when you do deploy it, you're going to be left wondering did I do so in an optimal way? Alternatively, because you safely stored the panel away months or years ago and never use it, it's more likely you'll forget all about it.

In short: at the moment you need the panel, you're going to wish you had experience with it.

Looking back at the video that sparked the comment, it's instructive to look at the various uses Common Sense Camper takes us through:

  • A Wood Carrier / Improvised Sink - this is a great way to learn about the construction, durability and physical limitations of the panel.
  • A Water Collector / First Aid Sling - rather than imagine other emergencies that this piece of gear could be used for, our reviewer actually tries them out.
  • As a Campsite Marker - this is a low risk way of using the panel as it was intended.

Each of these 'hacks' can be thought of as an exercise, the result of which will teach you volumes about this piece of gear. If the panel holds up to carrying wood, that goes a long way to showing just how durable it is. If the panel fails to help you find your campsite, then you're either using it wrong or it's not an effective tool for conditions. Regardless, it would be of infinite value to learn this information *before* a SAR team is tramping through the woods looking for you.

In short, respect rule #1: Know Thy Gear. And the sort of uses Common Sense Camper takes us through are great ways to do just this.

As for the original criticism raised by the commenter: yes, you'll want to regularly inspect the signal panel to confirm that it's in peak condition. But the same should be said of the knife, compass, and any other gear you plan to depend on out in the field. Arguably, taking the time to inspect the panel is a great way to put it top of mind before heading out into the woods.

Wednesday, December 26, 2018

Man vs. Nature -- And Nature Always Wins

These are photos from East Potomac Park, more or less around the corner from the Jefferson Memorial and the rest of sites on the DC Mall.

I'd say that this some sort of metaphor about rising sea waters, climate change, or the need to drain the swamp. But I fear that we're past the metaphoric stage, and into a literal one.

Pretty soon, I'm going to have to carry a life jacket when I go running in DC.

Friday, December 21, 2018

A Quick bash Morse Code Generator

I've gotten into the habit of scribbling down the standard Morse code alphabet on the last page of my pocket notepad:

I don't have a specific reason for this practice, I was just curious if it would come in handy. And as a matter of fact, earlier this week it did.

While visiting our nieces and nephew in Boston, we had a discussion about the Titanic that turned into a discussion about issuing an SOS, which in turn brought us around to Morse Code. Later in the evening, I found myself sitting with Dovid with a few minutes to kill while Shira and his Dad went through some diabetic maintenance tasks. I pulled out my notepad and gave Dovid a quick tutorial on how to encode and decode letters using Morse code. We took turns writing out a few short messages, and quickly ran out of time. I promised him that the next day I'd send him a longer message over e-mail for him to decode.

The next day, as I found myself thinking about the message I was going to send him, it occurred to me just how painful hand-translating the text was going to be. The programmer in me kicked in and I quickly whipped up the script below to map text to dots and dashes. Using it, I was able to do the following:

$ smorse 'Dovid, you rock!'
-.. --- ...- .. -.. , / -.-- --- ..-  / .-. --- -.-. -.- !

$ (echo "Dovid," ; echo "We had fun" ; echo "visiting with you" ; echo "yesterday") |smorse 
-.. --- ...- .. -.. ,
.-- .  / .... .- -..  / ..-.. ..- -. 
...- .. ... .. - .. -. --.  / .-- .. - ....  / -.-- --- ..- 
-.-- . ... - . .-. -.. .- -.-- 

While the command is noticeably slow, I'm impressed with how it came out. It's certainly much faster than hand translation.

Here's the code for the command. Who knows, maybe you too can use it to impress your niece or nephew.

#!/bin/bash                                                                                                                                                                                                         

##                                                                                                                                                                                                                  
## Very Simple Morse Code Translator                                                                                                                                                                                
##                                                                                                                                                                                                                  


input=$HOME/.smorse.input.$$
output=$HOME/.smorse.output.$$

mapping=$(cat <<EOF                                                                                                                                                                                           
a:.-                                                                                                                                                                                                                
b:-...                                                                                                                                                                                                              
c:-.-.                                                                                                                                                                                                              
d:-..                                                                                                                                                                                                               
e:.                                                                                                                                                                                                                 
f:..-..                                                                                                                                                                                                             
g:--.                                                                                                                                                                                                               
h:....                                                                                                                                                                                                              
i:..                                                                                                                                                                                                                
j:.---                                                                                                                                                                                                              
k:-.-                                                                                                                                                                                                               
l:.-..                                                                                                                                                                                                              
m:--                                                                                                                                                                                                                
n:-.                                                                                                                                                                                                                
o:---                                                                                                                                                                                                               
p:.--.                                                                                                                                                                                                              
q:--.-                                                                                                                                                                                                              
r:.-.                                                                                                                                                                                                               
s:...                                                                                                                                                                                                               
t:-                                                                                                                                                                                                                 
u:..-                                                                                                                                                                                                               
v:...-                                                                                                                                                                                                              
w:.--                                                                                                                                                                                                               
x:-..-                                                                                                                                                                                                              
y:-.--                                                                                                                                                                                                              
z:--..                                                                                                                                                                                                              
EOF                                                                                                                                                                                                                 
)

if [ -z "$*" ] ; then
  cat > $input
else
  echo "$@" > $input
fi

tr '[A-Z]' '[a-z]' < $input > $output
mv $output $input

sed 's| | / |g' < $input > $output
mv $output $input

for pair in  $mapping ; do
  letter=$(echo $pair | cut -d : -f 1)
  code=$(echo $pair | cut -d : -f 2)
  sed "s/$letter/$code /g" < $input > $output
  mv $output $input
 done

cat $input

Thursday, December 20, 2018

These Kids Get It -- Talking Web App Development With Fourth Graders

Yesterday I had the chance to talk about my life as a programmer and business owner with two fourth grade classes. It was fun! To liven things up, I walked the kids through an imaginary project: developing a web app to help parents pick the best summer vacation activity. The kids asked great questions and had no problem following along.

Here's a bunch of resources I put together to power my sample project:

  • A Google Form to capture input from students about their summer plan preferences. The students use Google Forms regularly, so this was known territory for them. My lesson to them: they can build software using existing tools they already know.
  • A Google Sheet to collect up data from the form. Here I demonstrated the use of conditional formatting. My lesson to them: some programs, like conditional formatting expressions, are tiny but quite powerful.
  • An XML data feed used to transfer data from the Google Sheet to my custom code. My lesson to them: behind the Internet you know is one designed to allow programs to talk to each other. You can use these connections to build something amazing.
  • A PHP app that pulled in the data, did some simple math and generated parent friendly output.
  • A Google Sheets add-on that generates test data. After highlighting that our system only works if there's data behind it, we talked about what a tedious job it would be to load data into the system. My lesson to them: if you find yourself doing the same thing over and over again, stop and write some code to do this.

Nearly everything about the presentation went to plan: I was able to plug my Chromebook into the projector, access the internet and allow the students to fill in the Google Form in real time. Heck, I found a way around the school's firewall to access ssh. Unfortunately, my presentation was a bit too authentic: when the time came for the big reveal of the web app, I realized that there was a bug that caused it to show the wrong results. D'oh!

The bug was easy to find a fix. But alas, with a room full of 60 fourth graders I couldn't debug the issue on the fly. Still, it was a teachable moment about the life of programmer.

All in all, it was a delightful day and a joy to talk shop with a group of smart kids.

Monday, December 17, 2018

Tea, Pottery and Black Bears - One Very Adventurous Day in Boston

Shira and I have made our share of quick trips to Boston to visit our nieces and nephew, but this last trip, clocking in at less than 35 hours, is probably our briefest. We essentially had one full day to play with the kids, and play we did. The weather, a cold and rainy 47°F, didn't do us any favors. But we remained undeterred!

Our first activity was the visit to the Boston Tea Party Museum. Because yesterday was the anniversary of the event, all tours were free. The kids were confused by the mock-town meeting that starts the tour, but certainly enjoyed going on the model ship. They each got to toss (and retrieve) a bundle of tea and take turns 'driving the ship' by manning the captain's wheel. We didn't spend much time on the boat, but given how nasty the weather was, nobody complained.

We then proceeded into a small series of rooms indoors, where we watched a number of cleverly structured videos.

I was struck by a number of things. First, the museum did its best to impart the gravity of the Tea party on us: at a minimum, the colonists were committing theft, at a maximum, treason. Second, the ship the colonists raided is tiny. OK, maybe not tiny, but when you consider that we were standing a on replica of a boat intended to sail across the ocean, it's remarkable how small it is. How little protection would it provide in open ocean during a storm? I can't imagine. And finally, it's truly baffling that our founding fathers could see their plight as a form of slavery, yet turn around and write slavery into the constitution? Oh to be so blind.

The kids, of course, took away a different experience. They no doubt enjoyed throwing tea overboard and were blown away by the "talking portraits."

The museum isn't especially large, which worked to our advantage. We had other activities planned for the day, so getting a compact dose of history was perfect.

After lunch we made our way to our next activity: Ceramics A La Carte. Each kid got to pick out a ceramic statue and paint it as he or she wanted to. Chana selected a kitty, Tzipora a guinea pig, Gavriella a princess and Dovid a fighter jet. After a few minutes of instructions and warnings, the kids got to work. I realized that if I didn't paint something myself, I'd spend all my time micromanaging the children. I grabbed a salt-and-pepper shaker and went to work.

This was fun! In many respects, it was like the glass project we did at the Corning Museum of Glass. However, there was one key difference: there was no rushing. We all got to take our time. The kids really did fantastic and didn't need my coaching. Of course, on the way home, I thought of half-a-dozen ways I'd tackle the project differently, but that's par for the course. What a great activity everyone could do on their own terms, and one where the weather couldn't throw us off.

After the pottery and some snacks, we made our way to our final activity of the day: ZooLights at the Stone Zoo. Without the benefit of daylight, and with a healthy dose of festive lights, we managed to give the kids a unique take on a trip to the zoo. Maybe I was taking the name too literally, but I wasn't especially blown away by the lighting. But the kids love seeing animals, and every time we'd come across an active enclosure, we'd all ooh and ahh with delight. While the little fuzzy monkeys may have been the cutest animals we saw, the most memorable were probably the black bears. The bear enclosure is setup so you can approach thick glass. We did just this and noticed a couple of bears lounging in the back of the area. Then, without a sound, one bear walked right in front of the glass and managed the surprise Dovid. It was priceless.

After about 45 minutes, the rain started coming down again and we called it a night. If it had been better weather, we'd probably have explored more. Though, the kids definitely got a kick out of seeing what they did.

After a pizza dinner, a round of kid-friendly charades, and a bedtime story, our day was complete. We went back to our hotel room where I collapsed in a heap. We may have had a single day, but man, did we make it count!

Wednesday, December 12, 2018

The SAR-Tech Sinnet | Easy Paracord Access

7 minutes into his What's in my 24 Hour SAR pack, Ephraim pulls out an impressively spooled length of florescent paracord. He explains that it's wrapped using the SAR Tech Sinnet method. The name, he explains, comes from the fact that this method of paracord storage is preferred by Canadian Armed Forces Search and Rescue technicians. Got to love those Canadians; even their paracord is polite.

A search for SAR-Tech Sinnet turned up this YouTube video which shows how to create the bundle. Further research shows that the rest of the world knows this method of storage as creating a paracord donut.

Regardless of what it's called, it's obviously handy and I'm amazed I haven't come across it before. Once the SAR-Tech Sinnet is created, it's trivial to spool off controlled lengths of paracord without it knotting. The donut itself can serve as a weight, which helps when tossing a length of paracord . Learning to create the sinnet takes only a few minutes, and involves little more than repeating a trivial pattern. The only downside seems to be that it's not an especially fast way to bundle cordage. However, I'll gladly put in 15 minutes of prep-work at home, to avoid 10 minutes of wrestling with a knotted hunk of rope in the cold and dark.

Here's a 25 foot sinnet I created in about 10 minutes:

If you've got lengths of rope lying around, you really have to give this technique a try.

Tuesday, December 11, 2018

The Algorithm That Saved the Union | Civil War Encryption

OK, maybe that title is a bit click-bait'y. But it's true that the algorithm that powered the North's encryption during the US Civil War no doubt had a significant impact on the outcome of the war. At the core of this algorithm was the use of a route cipher. This cipher is brilliant in its simplicity.

Start with a grid of spaces and a non-obvious route to visit each cell in the grid:

Fill grid with your message, putting one word per cell.

To encrypt the message, traverse the prescribed route, writing down each word as it is encountered:

Note the delightfully jumbled message.

To decode the message, traverse the route again, filling in words as you move from cell to cell. When the grid is fully traversed, the original message will be restored in the grid and can be read as clear text.

The Union Army strengthened this basic cipher with a number of enhancements:

  • Critical words were exchanged with code words. For example, any time the word enemy was to be used wiley was substituted.
  • Each route had a corresponding keyword that itself was included in the message. This allowed lengthy messages to be encrypted using a variety of routes, with the results being concatenated together.
  • Nonsense words could be added in the first and last row of the grid. This resulted in noise being mixed into with the message. When decrypted, the meaningless words were easy to discard.

You can see all these features at work in the example from the article Internal Struggle: The Civil War. The screenshots below show my coding of the routes, dictionary, clear text and coded message.

My implementation of the Union Cypher is on the clunky side. For one thing, I wanted to support more complex routes than simply walking up and down columns. The staton and mcdowell routes for example, call for traversing diagonally from the bottom left hand corner to the top right hand, and then proceeding column-wise. To support this, I describe routes by numbering each cell in the order they are to be visited. This makes for tedious route definition, but it is also quite flexible. You could imagine routes that had a checkerboard shape or other unusual patterns. I've also explicitly included the rows that will contain nonsense words in the grid.

These programming complications are great examples of steps in an algorithm that a human can trivially process but a machine needs to explicitly account for.

Just as interesting as the use of the cypher is the context within which it developed. The Civil War posed a unique military challenge:

The contending forces spoke the same language, shared the same social institutions, including an un-muzzled press and a tendency to express oneself freely on any subject. Military knowledge was shared in common--former classmates at the service academies and peacetime friends would meet in battle. They knew each other’s strengths and weaknesses, and they eagerly devoured reports, in the press and through intelligence sources, of the names of opposing commanders. Each harbored sympathizers with the other side, the basis for espionage and a potential fifth column. Neither inherited any competence in information security nor an appreciation for operational security. Those things would be learned the hard way--the American way--accompanied by bloodshed.

While the Union and Confederacy started from the same point, they made technological choices that ultimately drove how successful they would be in the world of information security. For example, the North's choice of word jumbling over a letter based cypher would have profound impact:

In the North, as telegraphers (frequently little more than teenage boys) were pressed into service and formed into the U.S. Military Telegraph (USMT), a rival of Myer’s signal corps, a word, or route, transposition system was adopted and became widespread. It gave the telegraphers recognizable words, an asset in this early stage of copying Morse “by ear,” that helped to reduce garbles. Code names or code words replaced sensitive plain text before it was transposed, and nulls disrupted the sense of the underlying message. Only USMT telegraphers were permitted to hold the system, thereby becoming cipher clerks as well as communicators for their principals, and the entire organization was rigidly controlled personally by the secretary of war. In the War Department telegraph office near the secretary, President Lincoln was a frequent figure from the nearby White House, anxiously hovering over the young operators as they went about their work.

In the South, although a Confederate States Military Telegraph was organized (in European fashion, under the Postmaster General), it was limited to supplementing the commercial telegraph lines. (“System” would not convey the proper idea, for the Southern lines were in reality a number of independent operations, some recently cut off from their northern ties by the division of the nation and reorganized as Southern companies.) Throughout the war, the Confederate government paid for the transmission of its official telegrams over commercial lines. Initially the Southern operator found peculiar digital texts coming his way (the dictionary system), then scrambled, meaningless letters, begging to be garbled. The poly-alphabetical cipher used for official cryptograms offered none of the easily recognizable words that provided a crutch for his Northern brother.

This is an interesting example of how an apparently less secure system (one where words are kept intact) can ultimately prove to be more valuable than a seemingly more secure one. Or, put another way: never underestimate the impact of human error.

Sunday, December 09, 2018

Puerto Rico Adventure - Day 7 - The Last Day

[Composed 11/26/2018]

When we awoke this morning we knew that by the end of the day we'd be getting on an airplane with its destination being a frigid Washington, DC. This alone motivated me to try to pack as many mini-adventures as possible into the day.

We started off with a tour through the Jardin Botanico UPR - Rio Piedras. The garden wasn't the largest or flashiest we'd ever been to. But was a wonderful way to get a walk in a natural setting without having to drive a significant distance. We saw a handful of interesting birds and flowers and generally enjoyed soaking up our last day of perfect Puerto Rican weather. Combine this with free-admission, and I'd rank this as a worthy place to visit should you find yourself in San Juan. On our way out, we asked about the orchids. The guide explained that Maria had decimated them and there were no longer any to be found in the park. Apparently the garden had taken quite a beating and the orchids were only one of the many victims. This was theme for us throughout our week in Puerto Rico: to the untrained eye, hurricane Maria had hadn't left much of a mark. But dig a little deeper and it seems like every aspect of Puerto Rican life was impacted by this hurricane, and there's still much left for the island to recover from.

After the gardens, we made our way to Plaza del Marcado de Rio Piedras. Like the botanic gardens, this market isn't anywhere near as epic as ones we've visited in Morocco or Japan. Still, it was was a fun market to walk through, and had we needed to pick up any cheap supplies this would have arguably been the place to do it.

We ate lunch at Fela's vegetarian restaurant, which was quite delicious. They offered quite an extensive set of veggie options, allowing us to create quite the veggie friendly smorgasbord.

We spent our final few hours in San Juan by wandering the Plaza Las Americas, which has the distinction of being the largest shopping mall in the Caribbean. When it opened, it was the largest shopping mall in Latin America. Yeah, it's big. But mostly, it felt like stepping back in time. What with Sears and JC Penney being anchor stores, and walking by a Time-Out Arcade.

After the mall, we fought traffic to get to the airport, where we boarded for an on-time departure.

We really can't recommend Puerto Rico highly enough. It's got everything we look for in a destination: hiking, history and lots of opportunity for adventure, and welcoming natives. The casinos don't hurt either.